Papers with conditional masked language model
Incorporating a Local Translation Mechanism into Non-autoregressive Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to capture local dependencies among output tokens are not efficient, causing errors of repeated translation. |
| Approach: | They propose a local autoregressive translation mechanism that predicts a short sequence of tokens for each target decoding position instead of one token. |
| Outcome: | Empirical results show that the proposed method achieves comparable or better performance with fewer decoding iterations, bringing a 2.5x speedup. |
Universal Conditional Masked Language Pre-training for Neural Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained sequence-to-sequence models have significantly improved Neural Machine Translation (NMT) this paper demonstrates that pre-training a sequence- to-squence model with a bidirectional decoder can produce notable performance gains for both Autoregressive and Non-autoregressive NMT tasks. |
| Approach: | They propose a conditional masked language model pre-trained on bilingual and monolingual corpora in many languages. |
| Outcome: | The proposed model can achieve significant performance improvements on all scenarios from low- to extremely high-resource languages. |
Con-NAT: Contrastive Non-autoregressive Neural Machine Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Neural machine translation models are autoregressive, which means they predict tokens one by one based on source tokens and previously predicted tokens. |
| Approach: | They propose a conditional masked language model which incorporates contrastive learning into the conditional language model. |
| Outcome: | The proposed model improves on WMT’16 Ro-En translation directions with different data sizes. |
Isotropy-Enhanced Conditional Masked Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing non-autoregressive models with auto-regressing decoding paradigms have been used for various text generation tasks to accelerate inference but at the cost of generation quality to some extent. |
| Approach: | They propose to use Look Neighbors strategy to enhance learning of target token representations during training to achieve a good balance between inference speedup and generation quality. |
| Outcome: | The proposed models outperform current models on 4 WMT datasets and outperformed the current SoTA results. |